Balancing Precision and Retention
in Experimental Design
Gustavo Diaz
Northwestern University
gustavo.diaz@northwestern.edu
gustavodiaz.org
Erin Rossiter
University of Notre Dame
erossite@nd.edu
erossiter.com
Paper and slides: gustavodiaz.org/talk
Bias-variance tradeoff as darts
But the game of darts is more complicated
Two types of tradeoffs
Explicit: Is some bias worth the increase in precision?
Implicit: Improving precision without sacrificing unbiasedness?
Two types of tradeoffs
Explicit: Is some bias worth the increase in precision
Implicit: Improving precision without sacrificing unbiasedness?
Cost has to come from somewhere else!
Improving precision in experiments
Standard error of estimated ATE in conventional experimental design (Gerber and Green 2012, p. 57)
\[ SE(\widehat{ATE}) = \sqrt{\frac{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}{N-1}} \]
Improving precision in experiments
\[ SE(\widehat{ATE}) = \sqrt{\frac{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}{N-1}} \]
Improving precision in experiments
\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{N-1}} \]
Variance component
Decrease \(SE(\widehat{ATE})\) with alternative research designs
Improving precision in experiments
\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{N-1}} \]
Variance component
Decrease \(SE(\widehat{ATE})\) with alternative research designs
Block-randomization
Repeated measures
Pre-treatment covariates
Pair-matched design
Online balancing
Sequential blocking
Rerandomization
Matching
Improving precision in experiments
\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{N-1}} \]
Variance component
Decrease \(SE(\widehat{ATE})\) with alternative research designs
Block-randomization
Repeated measures
Pre-treatment covariates
Pair-matched design
Online balancing
Sequential blocking
Rerandomization
Matching
Improving precision in experiments
\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{N-1}} \]
Variance component
Decrease \(SE(\widehat{ATE})\) with alternative research designs
Block-randomization
Repeated measures
Pre-treatment covariates
Pair-matched design
Online balancing
Sequential blocking
Rerandomization
Matching
All require pre-treatment information
Improving precision in experiments
\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{N-1}} \]
Variance component
Decrease \(SE(\widehat{ATE})\) with alternative research designs
Block-randomization
Repeated measures
Pre-treatment covariates
Pair-matched design
Online balancing
Sequential blocking
Rerandomization
Matching
All require pre-treatment information
Two categories:
Reduce variance in observed outcomes
Reduce variance in potential outcomes
Improving precision in experiments
\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{\color{#00843D}{N-1}}} \]
Sample size component
Improving precision in experiments
\[ SE(\widehat{ATE}) = \sqrt{\frac{\color{#4E2A84}{\text{Var}(Y_i(0)) + \text{Var}(Y_i(1)) + 2\text{Cov}(Y_i(0), Y_i(1))}}{\color{#00843D}{N-1}}} \]
Sample size component
Quadruple to halve \(SE(\widehat{ATE})\)
Focus: Increasing numerator may come at the cost of decreasing denominator
Precision gains from alternative designs may be offset by sample loss